2 |
On the Role of Corpus Ordering in Language Modeling ...
|
|
|
|
Abstract:
Language models pretrained on vast corpora of unstructured text using self-supervised learning framework are used in numerous natural language understanding and generation tasks. Many studies show that language acquisition in humans follows a rather structured simple-to-complex pattern and guided by this intuition, curriculum learning, which enables training of computational models in a meaningful order, such as processing easy samples before hard ones, has been shown to potentially reduce training time. The question remains whether curriculum learning can benefit pretraining of language models. In this work, we perform comprehensive experiments involving multiple curricula strategies varying the criteria for complexity and the training schedules. Empirical results of training transformer language models on English corpus and evaluating it intrinsically as well as after fine-tuning across eight tasks from the GLUE benchmark, show consistent improvement gains over conventional vanilla training. Interestingly, ...
|
|
URL: https://underline.io/lecture/39768-on-the-role-of-corpus-ordering-in-language-modeling https://dx.doi.org/10.48448/0ea2-mc46
|
|
BASE
|
|
Hide details
|
|
3 |
Evaluation of Unsupervised Automatic Readability Assessors Using Rank Correlations ...
|
|
|
|
BASE
|
|
Show details
|
|
4 |
Analysis of Language Change in Collaborative Instruction Following ...
|
|
|
|
BASE
|
|
Show details
|
|
5 |
Learning Feature Weights using Reward Modeling for Denoising Parallel Corpora ...
|
|
|
|
BASE
|
|
Show details
|
|
6 |
Cross-lingual Aspect-based Sentiment Analysis with Aspect Term Code-Switching ...
|
|
|
|
BASE
|
|
Show details
|
|
7 |
Cross-lingual Transfer for Text Classification with Dictionary-based Heterogeneous Graph ...
|
|
|
|
BASE
|
|
Show details
|
|
8 |
NOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Dataset ...
|
|
|
|
BASE
|
|
Show details
|
|
10 |
An Unsupervised Method for Building Sentence Simplification Corpora in Multiple Languages ...
|
|
|
|
BASE
|
|
Show details
|
|
12 |
SD-QA: Spoken Dialectal Question Answering for the Real World ...
|
|
|
|
BASE
|
|
Show details
|
|
13 |
Plan-then-Generate: Controlled Data-to-Text Generation via Planning ...
|
|
|
|
BASE
|
|
Show details
|
|
14 |
Sparsity and Sentence Structure in Encoder-Decoder Attention of Summarization Systems ...
|
|
|
|
BASE
|
|
Show details
|
|
15 |
Identity-Based Patterns in Deep Convolutional Networks: Generative Adversarial Phonology and Reduplication ...
|
|
|
|
BASE
|
|
Show details
|
|
16 |
Live Session - 4E: Phonology, Morphology and Word Segmentation ...
|
|
|
|
BASE
|
|
Show details
|
|
19 |
Rule-based Morphological Inflection Improves Neural Terminology Translation ...
|
|
|
|
BASE
|
|
Show details
|
|
20 |
Translating Headers of Tabular Data: A Pilot Study of Schema Translation ...
|
|
|
|
BASE
|
|
Show details
|
|
|
|